Statistical Properties of Probabilistic Context-Free Grammars

We prove a number of useful results about probabilistic context-free grammars (PCFGs) and their Gibbs representations. We present a method, called the relative weighted frequency method, to assign production probabilities that impose proper PCFG distributions on finite parses. We demonstrate that these distributions have finite entropies. In addition, under the distributions, sizes of parses have finite moment of any order. We show that Gibbs distributions on CFG parses, which generalize PCFG distributions and are more powerful, become PCFG distributions if their features only include frequencies of production rules in parses. Under these circumstances, we prove the equivalence of the maximum-likelihood (ML) estimation procedures for these two types of probability distributions on parses. We introduce the renormalization of improper PCFGs to proper ones. We also study PCFGs from the perspective of stochastic branching processes. We prove that with their production probabilities assigned by the relative weighted frequency method, PCFGs are subcritical, i.e., their branching rates are less than one. We also show that by renormalization, connected supercritical PCFGs become subcritical ones. Finally, some minor issues, including identifiability and approximation of production probabilities of PCFGs, are discussed.